IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit based on Analyses of Interestingness

In recent years, advances in deep learning have resulted in a plethora of successes in the use of reinforcement learning (RL) to solve complex sequential decision tasks with high-dimensional inputs. However, existing systems lack the necessary mechanisms to provide humans with a holistic view of their competence, presenting an impediment to their adoption, particularly in critical applications where the decisions an agent makes can have significant consequences. Yet, existing RL-based systems are essentially competency-unaware in that they lack the necessary interpretation mechanisms to allow human operators to have an insightful, holistic view of their competency. Towards more explainable Deep RL (xDRL), we propose a new framework based on analyses of interestingness. Our tool provides various measures of RL agent competence stemming from interestingness analysis and is applicable to a wide range of RL algorithms, natively supporting the popular RLLib toolkit. We showcase the use of our framework by applying the proposed pipeline in a set of scenarios of varying complexity. We empirically assess the capability of the approach in identifying agent behavior patterns and competency-controlling conditions, and the task elements mostly responsible for an agent's competence, based on global and local analyses of interestingness. Overall, we show that our framework can provide agent designers with insights about RL agent competence, both their capabilities and limitations, enabling more informed decisions about interventions, additional training, and other interactions in collaborative human-machine settings.Comment: To be published in the Proceedings of the 1st World Conference on eXplainable Artificial Intelligence (xAI 2023). arXiv admin note: substantial text overlap with arXiv:2211.0637

Learning to integrate reactivity and deliberation in uncertain planning and scheduling problems

This paper describes an approach to planning and scheduling in uncertain domains. In this approach, a system divides a task on a goal by goal basis into reactive and deliberative components. Initially, a task is handled entirely reactively. When failures occur, the system changes the reactive/deliverative goal division by moving goals into the deliberative component. Because our approach attempts to minimize the number of deliberative goals, we call our approach Minimal Deliberation (MD). Because MD allows goals to be treated reactively, it gains some of the advantages of reactive systems: computational efficiency, the ability to deal with noise and non-deterministic effects, and the ability to take advantage of unforseen opportunities. However, because MD can fall back upon deliberation, it can also provide some of the guarantees of classical planning, such as the ability to deal with complex goal interactions. This paper describes the Minimal Deliberation approach to integrating reactivity and deliberation and describe an ongoing application of the approach to an uncertain planning and scheduling domain

Outcome-Guided Counterfactuals for Reinforcement Learning Agents from a Jointly Trained Generative Latent Space

We present a novel generative method for producing unseen and plausible counterfactual examples for reinforcement learning (RL) agents based upon outcome variables that characterize agent behavior. Our approach uses a variational autoencoder to train a latent space that jointly encodes information about the observations and outcome variables pertaining to an agent's behavior. Counterfactuals are generated using traversals in this latent space, via gradient-driven updates as well as latent interpolations against cases drawn from a pool of examples. These include updates to raise the likelihood of generated examples, which improves the plausibility of generated counterfactuals. From experiments in three RL environments, we show that these methods produce counterfactuals that are more plausible and proximal to their queries compared to purely outcome-driven or case-based baselines. Finally, we show that a latent jointly trained to reconstruct both the input observations and behavioral outcome variables produces higher-quality counterfactuals over latents trained solely to reconstruct the observation inputs

Confidence Calibration for Systems with Cascaded Predictive Modules

Existing conformal prediction algorithms estimate prediction intervals at target confidence levels to characterize the performance of a regression model on new test samples. However, considering an autonomous system consisting of multiple modules, prediction intervals constructed for individual modules fall short of accommodating uncertainty propagation over different modules and thus cannot provide reliable predictions on system behavior. We address this limitation and present novel solutions based on conformal prediction to provide prediction intervals calibrated for a predictive system consisting of cascaded modules (e.g., an upstream feature extraction module and a downstream regression module). Our key idea is to leverage module-level validation data to characterize the system-level error distribution without direct access to end-to-end validation data. We provide theoretical justification and empirical experimental results to demonstrate the effectiveness of proposed solutions. In comparison to prediction intervals calibrated for individual modules, our solutions generate improved intervals with more accurate performance guarantees for system predictions, which are demonstrated on both synthetic systems and real-world systems performing overlap prediction for indoor navigation using the Matterport3D dataset

Interactive and Mixed-Initiative Decision Theoretic Systems: Papers from the Adaptive Mixed-Initiative Systems for Decision-Theoretic Crisis Response

ding and its occupants, and so on. Gathering more information may reduce this uncertainty, but in the time this takes, the magnitude and the scope of the problem presented by the spill will continue to increase. Decision theory seems a natural approach, given the characteristics of crisis. However, decision-theoretic techniques are particularly difficult to implement for crisis domains for the following reasons. Decisiontheoretic methods require the development of complete models, a time-consuming task ill-afforded in crisis domains with their characteristic of urgency. The limited amount of time for decision-making also constrains the development of alternative courses of action and their subsequent consideration. Furthermore, the uncertainty inherent in crisis domains makes the enumeration of alternatives as well as the elicitation of probability models more difficult. However, decision-theoretic methods remain attractive for crisis domains because they provide wellfounded principle

Adapting to User Preferences in Crisis Response

The domain of crisis planning and scheduling taxes human response managers due to high levels of urgency and uncertainty. Such applications require assistant technologies (in contrast to automation technologies) and provide special challenges for interface design. We present INCA, the INteractive Crisis Assistant, that helps users develop effective crisis response plans and schedules in a timely manner. INCA also adapts to the individual users by anticipating their preferred responses to a given crisis and their intended repairs to a candidate response. We evaluate our system in HAZMAT, a synthetic hazardous materials incident domain. The results show that INCA tailors itself to individual users and provides effective support for the timely generation of effective responses

An incremental curative learning approach for planning with incorrect domain theories

Incorrect domain theories, and the flawed plans derived from them, are an inescapable aspect of planning in the real world. Previous learning approaches to the incorrect domain theory problem have relied on diagnosing failures to determine fixes to the domain theory to prevent the failures in the future. However, perfect failure diagnosis is itself an intractable problem, and preventive learners are subject to the problem of faulty fixes due to incorrect failure diagnoses.This thesis presents completable planning, an incremental learning approach that augments classical plans with generalized alternative plan segments called contingent plans, which serve to cure failures by providing recovery routes from the execution failures of flawed plans constructed using incorrect domain theories. By constructing contingent plans only in response to actual failures encountered during execution, a completable planner is saved the effort of planning for all possible outcomes. By requiring only that failures be treated and not necessarily explained, a completable planner is freed from the cost and reliance on perfect failure diagnosis. The contingent plans learned through completable planning are also guaranteed never to decrease the probability of success of the original plan.A completable planning strategy is defined by the decisions on the expectations to verify during execution, when to initiate contingent planning in response to an unexpected outcome, and what portion of the plan to reuse. As a curative approach, completable planning is more susceptible than a preventive approach to the limitations of its native planning strategy. An investigation of the effects of completable planning strategy on planning performance and learning performance is thus important, and this thesis presents the results of such a study.This thesis also presents a method for addressing these limitations called probably approximately correct (PAC) completable planning. The PAC approach to completable planning allows a completable plans to find one that satisfies given constraints on the probability of success of a plan, execution cost, and planning cost.U of I OnlyETDs are only available to UIUC Users without author permissio